Learning in Hybrid Noise Environments Using Statistical Queries

We consider formal models of learning from noisy data. Speciically, we focus on learning in the probability approximately correct model as deened by Valiant. Two of the most widely studied models of noise in this setting have been classiication noise and malicious errors. However, a more realistic model combining the two types of noise has not been formalized. We deene a learning environment based on a natural combination of these two noise models. We rst show that hypothesis testing is possible in this model. We next describe a simple technique for learning in this model, and then describe a more powerful technique based on statistical query learning. We show that the noise tolerance of this improved technique is roughly optimal with respect to the desired learning accuracy and that it provides a smooth tradeoo between the tolerable amounts of the two types of noise. Finally, we show that statistical query simulation yields learning algorithms for other combinations of noise models, thus demonstrating that statistical query speciication truly captures the generic fault tolerance of a learning algorithm. An important goal of research in machine learning is to determine which tasks can be automated, and for those which can, to determine their information and computation requirements. One way to answer these questions is through the development and investigation of formal models of machine learning which capture the task of learning under plausible assumptions. In this work, we consider the formal model of learning from examples called \probably approximately correct" (PAC) learning as deened by Valiant Val84]. In this setting, a learner attempts to approximate an unknown target concept simply by viewing positive and negative examples of the concept. An adversary chooses, from some speciied function class, a hidden f0; 1g-valued target function deened over some speciied domain of examples and chooses a probability distribution over this domain. The goal of the learner is to output in both polynomial time and with high probability, an hypothesis which is \close" to the target function with respect to the distribution of examples. The learner gains information about the target function and distribution by interacting with an example oracle. At each request by the learner, this oracle draws an example randomly according to the hidden distribution, labels it according to the hidden target function, and returns the labelled example to the learner. A class of functions F is said to be PAC learnable if 1996 Springer-Verlag.